Performance Tuning

Disk IO

Achieving high throughput transfers requires first and foremost that you have the physical ability to read/write to data at the speeds you are trying to transfer.

FileCatalyst HotFolder and Remote Server Admin now include features that allow you to test write speed of your system. Running a file IO test can help determine if your system has adequate bandwidth to write the data you are transferring to it.

If the IO speed is not what you expected, verify the following:

1. Ensure that the user home directory (or HotFolder location) is pointing to the correct path and permission to read/write to the storage.
2. If the mount point is a network drive (Samba, SAN), make sure that the networking infrastructure can support the capacity (i.e, Trying to use a 1 Gbps ISCSI mounted via a 100 Mbps switch)

Attaining disk speeds for >1 Gbps transfers

For attaining transfer speeds higher than 500 Mbps on a single connection, additional tools are included with FileCatalyst to help you tune your IO speeds. Two scripts exist (one that performs read IO tests, the other performs write IO tests), and both can be launched via command-line by the Server/HotFolder/CLI Client JAR files. Here is an example using the CLI JAR file:f

java -jar FileCatalystCL.jar -testReadIO

java -jar FileCatalystCL.jar -testIO
              

Note: The same command works with FileCatalystServer.jar or FileCatalystHotFolder.jar.

The scripts are aimed at profiling your particular system, and testing to see if there are additional settings which can improve performance. The scripts require you to answer series of questions to set up the test parameters. Here is an example output for the supplied WRITE test script in Windows:

C:\tmp>java -jar FileCatalystCL.jar -testIO
Entering Write TestIO. This will run a series of tests on the file system to
attempt to discover the optimal values for # of writer threads and write block
size for your system.The test is both IO intensive and CPU intensive. Please
give the test adequate time to complete.

Please enter the drive/path you wish to test (ie:  C:/ or /mnt/data/ ):  c:/temp
  File to be written:  c:/temp/test.io

Please enter the size of file you wish to write (in MB, default 500):  10000
  File size:  10000MB.

Please enter the timeout length (secs) per run (default 60 secs): 180
  Timeout:  180 seconds.

Please enter the number of runs to perform for each setting (default 5): 3
  Number of runs per iteration:  3

Test if buffer size used in writes to disk affect performance.
Please enter a buffer array to attempt (default:  '64,128,256,512,1024,2048,4096')
Size in KB, comma delimited: 4,16,64,256,1024,4096,16384
  8 Buffer Size values (KB):  4,16,64,256,1024,4096,16384.

Test if multiple writers offer performance benefit when saving a block to disk.
Please enter a writer thread array to attempt (default:  '1,2,4,6,8'): 1,2
  2 Thread values:  1,2.

How many files would you like to create concurrently for IO test (default 1)?
Note: The number of files will never exceed the number of writer threads during tests.
  1 
  Files will be created in the test:  1.

Test using Direct IO when allocating buffer space by Java (default true):  true
  Use DirectIO = true.

Mode used to open up files (rw/rws/rwd -- default rw):  rw
  Mode used = rw.
                

Tests run with the following parameters:
        file:  /opt/tmp/test.io
        size:  10000000000
        timeout:  60000
        directIO:  true
        file mode:  rw
        Max # files to use:  1
  # of THREADS  |1      |2
Buffer size     +=======+=======+
4               |1431   |1282
16              |1722   |1614
64              |2059   |1748
256             |2239   |1933
1024            |2095   |2050
4096            |2078   |2048
16384           |1841   |1720
                

This test, for example, gives us a good indication that a read block size between 256-1024 KB works well, and that additional reader threads do not necessarily give a write performance boost. Note that if you are using on-the-fly compression, additional writer threads may help offload high CPU tasks to multiple cores on your machine.

Encryption and CPU Overhead

Certain features of the FileCatalyst Software are computationally intensive, and may bottleneck high-speed transfers.

Encryption of the Control Channel alone does not impose significant overhead. However, setting the FileCatalyst Server to use encryption on the Control and Data Channels do impose a performance (CPU) penalty.

For most clients, a typical client or server machine can handle transfer speeds of 100 Mbps. However, hitting higher speeds (>500 Mbps) becomes difficult if the Data Channel is encrypted, with transfer starting to become CPU-bound. This may be verified by running an Activity Monitor (i.e. "Task Manager" on Windows or "top" command on Linux) for your OS and checking the CPU usage while the transfer is occurring.

NOTE: AES Data Encryption is performed by the Sender threads. Increasing the # of sender threads to the # of cores on your machine allows the workload to be divided amongst multiple cores. AES Data Decryption is performed by Packet Processors. A modern processor can handle approximately 200-500 Mbps of encryption. It is recommended that the number of Packet Processors are increased per 500 Mbps you want to achieve in throughput for a single connection, or as many cores as your system has.

Depending on your workflow, another possible solution to overcome this would be to encrypt the data with an external application or process, before transferring it over the wire, and decrypt it once the data has reached its destination.

Compressing files can also be CPU intensive. Under most circumstances, the gain in network throughput outweighs the CPU overhead to zip up files before transfer, especially when files are text-based or other formats that can be highly compressed. However, compression should be avoided for most binary and media files, which are normally considered non-compressible. If compression and AES on the Data Channel are utilized, high-speed transfers are likely to be limited by the amount of computing power that can be assigned for each transfer.

Windows UDP packet limits for transmitter

Microsoft Windows uses different algorithms to process UDP packets based on the packet size. By default, packets of size 1024 or smaller use a fast method (pushed onto the network immediately), while larger packets (1025 or larger) are handled using a slower interrupt method which consumes less CPU (see: http://support2.microsoft.com/kb/235257 for more details). While these defaults are in place, if you configure Windows to use IP standard MTU 1500 byte packets, performance is significantly degraded. For this reason, FileCatalyst software limits packet size to 1024 bytes for all Windows senders.

You can, however, get faster performance on Windows by setting a Windows registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\AFD\Parameters\FastSendDatagramThreshold Setting this registry key tells Windows to use the fast method of sending UDP packets for larger packets as well. There is a script (enableFastDatagram.reg) included in the FileCatalyst application installation directory which can be run (requires administrative privileges to run and a reboot to take effect) to move the threshold from 1024 bytes to 2048 bytes. For optimum performance, it is recommended to run this script, then increase packet sizes to 1472 in FileCatalyst products on Windows platforms (provided the network supports standard MTU of 1500 packets).

Running the script undoFastDatagram.reg will remove the registry entry (also requires administrative privileges and a reboot).

Compress Text Files

FileCatalyst clients allow files to be compressed before being sent out. Transferring of text files (logs, emails, etc) should be compressed before being transferred. The CPU overhead of compression is negligible compared to the bandwidth savings of compressing an ASCII based file. It is recommended that Progressive Transfers are enabled when compression is used. This allows the file to be sent over even as it is being compressed, reducing the overall transfer time.

Media files (audio, video, images) however do not compress easily, and often add intensive computing overhead to a transfer while giving no bandwidth savings. Ensure that non-compressible files are filtered out by using the file compression filter.

Delta File Transfer

If the same file is sent out more than once (perhaps with only a small modification to the file), enabling Incremental Delta Transfers will significantly reduce the bandwidth involved in sending a file.

For large files, it is also recommended to turn on Progressive Transfers. This allows the file deltas to be transferred, even as they are being built on the FileCatalyst Server, reducing the time it takes for the file to be transferred.

If the same file contains entirely new data each time (example: logs truncated on daily basis, and uploaded using the same file name), it is recommended to disable incremental transfers and simply start to transfer the file as new (avoid the computation required to determine what is different in the file, as all of it has changed).

Disable Force Flush

By default, write operations automatically flush both data and metadata to file system for all IO calls. Disabling force flush allows Java to utilizes the Operating System's file cache (RAM).

Growing Files

In situations where growing files need to be transferred, it is recommended that Progressive Transfers are enabled to improve overall performance. By enabling this option for a transfer, the FileCatalyst application transfers the file while it grows on the file system. Once the individual transfer completes, the application will monitor the files to see if they change in size within a given time threshold. Should the file change within this threshold, the modifications will be transferred, and the application will return to monitoring the file system for new changes.

The following option for Dynamic Files increases performance by consistently transferring the file contents while it is being built. By transferring the files as they grow, the amount of time that is needed for the completed file to be present on the destination is reduced.

For more detailed information regarding Progressive Transfers, visit the help documentation for the FileCatalyst HotFolder application

Transfer Cache

If your data is constructed in a manner where new files are continuously being added into existing file set, it is recommended that you enable the Transfer Cache feature for your transfers. By enabling this option, transfers will only upload/download files that haven't already been sent or modified since they were last transferred by the application. This option reduces the amount of data being transferred, by making it so only changed or new files, are transferred by the system.

Transferring A Large Collection Of Small Files

Each file transfer, regardless of the file size, requires overhead to set up a network connection between the client and the server. For large files, this overhead has minimal impact, as the software will spend most of the time transferring data across the wire. For transfers of small individual files, however, the reverse is true: the software can spend most of the time setting up and tearing down a connection, only to transfer a directory full of many small files. To help reduce the connection overhead, there are two main ways that you can improve performance for a given transfer. The first way to enable Single Archive Compression of small files and the second option is to use Multi-Client instead of Single-Client for transfers.

By enabling the ‘the Single Archive Compression’ option, the application will compile all available small files into a single zip archive before it is transmitted. By performing this operation, the application now spends all its time transferring data instead of managing setup and tear down of network connections. Enabling Progressive Transfers will also allow the application to start transferring while the single archive is being built.

• Single Archive Note 1: There are caveats to Single Archive Compression transfers to keep in mind. Should a transfer be interrupted, auto-resume will not work, and restarting the transfer will start the archive and transfer from the beginning. Also, Delta File Transfers (incremental mode) does not work when Single Archive transfers are enabled.
• Single Archive Note 2: It is also recommended to enable the "Max size" feature for Single Archives. This will create several smaller archives and extract them as they are transferred. If the transfer is interrupted, generally some of the files will have already been transferred. The maximum value for the Max Size parameter should not exceed 3 GB for any Single Archive.

You may also improve the performance of transferring large collections of small files by enabling the Multi-Client option available in client applications. In a normal transfer, a Single-Client may only transfer a single file at a given time, which means the application must sequentially process the connection overhead for every file transferred. Multi-Client is different in that it able to concurrently transfer files with each client that it has available. By transferring files concurrently, the connection overhead for files is also processed concurrently across the clients, which in turn, reduces the total wait time that a transfer experiences when it is setting up its connections.

• Multi-Client Note: Multi-Client also supports an option titled "Auto-Archiving". Similar to the behaviour found in "Single Archive with Compression", Auto-Archiving takes small files less than 10 MB and compiles them into a zip archive with max size of 250 MB, before it is transferred. This option improves performance in large filesets with small file sizes by reducing the number of times the connection overhead is triggered.

Low Bandwidth Clients

For clients connecting with low bandwidth, ensure that the Congestion Control start rate is set below the maximum line speed. By default, the value for the start rate is 384 Kbps. If the Congestion Control rate is still higher than the available bandwidth, the application will quickly flood the network connection, producing lost/dropped packets and forcing the sender threads to perform most of its transfer in retransmission mode (slower).

For very low bandwidth clients and low latency transfers, FTP mode can at times perform faster transfer rates than UDP.

Using our online comparison calculator, you can see if there is an advantage to using FTP rather than UDP. Note, however, that the values in the online calculator do not take high packet loss into account. TCP at high latency is extremely sensitive to even 0.1% packet loss, while our UDP protocol handles this much more gracefully.

High Load Servers (or many clients)

Transferring UDP at high speed is memory intensive. Using the example listed above, a 250 ms transfer (LA to Hong Kong) at 100 000 Kbps requires a minimum of 25 MB of data to be in-flight at any one time.

For a standard install of FileCatalyst Server the default memory allocated is 1 GB. This means the system can likely accept around 40 concurrent connections at full speed before starting to run short of memory resources.

The key to getting the FileCatalyst Server, to accept higher numbers of concurrent connections is to lower the memory footprint each connection can take on the system. The FileCatalyst Server has an option via the Server Remote Administration to optimize for concurrency. This will lower the maximum thread count and block size each client connection consumes, resulting in the server able to accept more clients at any one time.

If the maximum sender thread and maximum block size are set too low, limiting clients from connecting at full bandwidth, the following may be done:

Limiting Concurrent Connections

The FileCatalyst Server allows setting the Maximum Concurrent User Connections to levels below what the license allows. This allows administrators the possibility of setting the memory footprint high for each client, but limiting the connections to ensure that the FileCatalyst Server does not itself start to run short of memory.

Increase Java Heap Size

The optimizations performed for high-load servers are based upon your existing available JAVA heap size. By increasing the memory that Java has access to, the number of threads or block sizes each transfer that may use is increased, which can increase the performance of the application under certain circumstances.

Windows Service

For FileCatalyst HotFolders running as a service, these may be configured in fchf.conf:

# Java Additional Parameters
# original: wrapper.java.additional.1=-XX:MaxDirectMemorySize=1024M
wrapper.java.additional.1=-XX:MaxDirectMemorySize=1536M

# Initial Java Heap Size (in MB)
# original: wrapper.java.initmemory=1024
wrapper.java.initmemory=2048

# Maximum Java Heap Size (in MB)
# original: wrapper.java.maxmemory=1024
wrapper.java.maxmemory=2048

Note: When using a 32-bit verison of Java, the memory available to a Java service can be increased to 1.5 GB (1536). You cannot currently modify the heap size in the Windows executable found in the Start Menu.

Linux Command Line

For FileCatalyst HotFolders started at the command line, the starting scripts may be modified to use alternative memory sizes than the standard 1GB:

fc_hotfolder.sh

# original: java -Xms64M -Xmx1024M -XX:MaxDirectMemorySize=512M -jar FileCatalystHotFolder.jar
-controlpanel & java -Xms256M -Xmx2048M -XX:MaxDirectMemorySize=1024M -jar FileCatalystHotFolder.jar 
-controlpanel &

Linux Service

For FileCatalyst HotFolders running as a service in Linux, these may be set in <application directory>/conf/wrapper.conf

# Java Additional Parameters
# original: wrapper.java.additional.1=-XX:MaxDirectMemorySize=1G
wrapper.java.additional.1=-XX:MaxDirectMemorySize=1536M

# Initial Java Heap Size (in MB)
# original: wrapper.java.initmemory=1024
wrapper.java.initmemory=1536

# Maximum Java Heap Size (in MB)
# original: wrapper.java.maxmemory=1024
wrapper.java.maxmemory=2048

NOTE: When using a 32-bit verison of Java, the memory available to a Java service can be increased to 1.5 GB (1536). In 64-bit versions of Java, the memory can be increased to a maximum of 4 GB (4096)